A Least-Squares Unified View of PCA, LDA, CCA and Spectral Graph Methods

Over the last century Component Analysis (CA) methods such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), Canonical Correlation Analysis (CCA) and Spectral Clustering (SC) have been extensively used as a feature extraction step for modeling, classification, visualization, and clustering. This paper proposes a unified framework to formulate PCA, LDA, CCA, and SC as a least-squares estimation problem. We show how these methods correspond to a particular instance of a weighted kernel reduced rank regression (WKRRR). The least-squares formulation allows better understanding of normalization factors, and provides an easier generalization of CA techniques. In particular, we derive the matrix expressions for weighted generalizations of PCA, LDA, SC and CCA (including kernel extensions), and show its effectiveness on synthetic and real problems. Finally, we suggest an efficient numerical method to solve WKRRR.